Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.

Can continuous diffusion models bring the same performance breakthrough on natural language they did for image generation? To circumvent the discrete nature of text data, we can simply project tokens in a continuous space of embeddings, as is standard in language modeling. We propose Self-conditioned Embedding Diffusion, a continuous diffusion mechanism that operates on token embeddings and allows to learn flexible and scalable diffusion models for both conditional and unconditional text generation. Through qualitative and quantitative evaluation, we show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models - while being in theory more efficient on accelerator hardware at inference time. Our work paves the way for scaling up diffusion models for text, similarly to autoregressive models, and for improving performance with recent refinements to continuous diffusion.

现实世界中的数据是高维的：即使在压缩后，书籍，图像或音乐表演也很容易包含数十万个元素。但是，最常用的自回归模型，变压器非常昂贵，以缩放捕获这种远程结构所需的输入和层数。我们开发了感知者AR，这是一种自回归的模态 - 不合骨架构，它使用交叉注意力将远程输入映射到少数潜在的潜在，同时还可以维护端到端的因果关系掩盖。感知器AR可以直接进行十万个令牌，从而实现了实用的长篇小写密度估计，而无需手工制作的稀疏模式或记忆机制。当对图像或音乐进行培训时，感知器AR会生成具有清晰长期连贯性和结构的输出。我们的架构还获得了长期基准测试的最新可能性，包括64 x 64个Imagenet图像和PG-19书籍。

Videos are a rich source of multi-modal supervision. In this work, we learn representations using self-supervision by leveraging three modalities naturally present in videos: visual, audio and language streams. To this end, we introduce the notion of a multimodal versatile network -a network that can ingest multiple modalities and whose representations enable downstream tasks in multiple modalities. In particular, we explore how best to combine the modalities, such that fine-grained representations of the visual and audio modalities can be maintained, whilst also integrating text into a common embedding. Driven by versatility, we also introduce a novel process of deflation, so that the networks can be effortlessly applied to the visual data in the form of video or a static image. We demonstrate how such networks trained on large collections of unlabelled video data can be applied on video, video-text, image and audio tasks. Equipped with these representations, we obtain state-of-the-art performance on multiple challenging benchmarks including UCF101, HMDB51, Kinetics600, Audioset and ESC-50 when compared to previous self-supervised work. Our models are publicly available [1, 2, 3]. * Equal contribution. † Work done during an internship at DeepMind. 34th Conference on Neural Information Processing Systems (NeurIPS 2020),

近期对抗性生成建模的突破导致了能够生产高质量的视频样本的模型，即使在真实世界视频的大型和复杂的数据集上也是如此。在这项工作中，我们专注于视频预测的任务，其中给出了从视频中提取的一系列帧，目标是生成合理的未来序列。我们首先通过对鉴别器分解进行系统的实证研究并提出产生更快的收敛性和更高性能的系统来提高本领域的最新技术。然后，我们分析发电机中的复发单元，并提出了一种新的复发单元，其根据预测的运动样本来改变其过去的隐藏状态，并改进它以处理DIS闭塞，场景变化和其他复杂行为。我们表明，这种经常性单位始终如一地优于以前的设计。我们的最终模型导致最先进的性能中的飞跃，从大型动力学-600数据集中获得25.7的测试集Frechet视频距离为25.7，下降到69.2。

This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-ofthe-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

Efficient and robust control using spiking neural networks (SNNs) is still an open problem. Whilst behaviour of biological agents is produced through sparse and irregular spiking patterns, which provide both robust and efficient control, the activity patterns in most artificial spiking neural networks used for control are dense and regular -- resulting in potentially less efficient codes. Additionally, for most existing control solutions network training or optimization is necessary, even for fully identified systems, complicating their implementation in on-chip low-power solutions. The neuroscience theory of Spike Coding Networks (SCNs) offers a fully analytical solution for implementing dynamical systems in recurrent spiking neural networks -- while maintaining irregular, sparse, and robust spiking activity -- but it's not clear how to directly apply it to control problems. Here, we extend SCN theory by incorporating closed-form optimal estimation and control. The resulting networks work as a spiking equivalent of a linear-quadratic-Gaussian controller. We demonstrate robust spiking control of simulated spring-mass-damper and cart-pole systems, in the face of several perturbations, including input- and system-noise, system disturbances, and neural silencing. As our approach does not need learning or optimization, it offers opportunities for deploying fast and efficient task-specific on-chip spiking controllers with biologically realistic activity.

Hamiltonian Monte Carlo（HMC）是Markov链算法，用于从具有密度$ e^{ - f（x）} $的高维分布中进行采样，可访问$ f $的梯度。一种特殊的感兴趣的情况是带有协方差矩阵$ \ sigma $的$ d $二维高斯分布，在这种情况下$ f（x）= x^\ top \ top \ sigma^{ - 1} x $。我们表明，HMC可以使用$ \ wideTilde {o}（\ sqrt {\ kappa} d^{1/4} \ log（1/\ varepsilon），使用$ \ varepsilon $ -close在总变化距离中取样。）$渐变查询，其中$ \ kappa $是$ \ sigma $的条件号。我们的算法对哈密顿动力学使用了长时间和随机的整合时间。这与最近的结果（并受到了）的形成对比，该结果给出了$ \ widetilde \ omega（\ kappa d^{1/2}）$查询的HMC较低限制，即使是高斯案例，也有固定的集成时间。

由于结构化数据通常不足，因此在开发用于临床信息检索和决策支持系统模型时，需要从电子健康记录中的自由文本中提取标签。临床文本中最重要的上下文特性之一是否定，这表明没有发现。我们旨在通过比较荷兰临床注释中的三种否定检测方法来改善标签的大规模提取。我们使用Erasmus医疗中心荷兰临床语料库比较了基于ContextD的基于规则的方法，即使用MEDCAT和（Fineted）基于Roberta的模型的BilstM模型。我们发现，Bilstm和Roberta模型都在F1得分，精度和召回方面始终优于基于规则的模型。此外，我们将每个模型的分类错误系统地分类，这些错误可用于进一步改善特定应用程序的模型性能。在性能方面，将三个模型结合起来并不有益。我们得出的结论是，尤其是基于Bilstm和Roberta的模型在检测临床否定方面非常准确，但是最终，根据手头的用例，这三种方法最终都可以可行。

城市规划师越来越多地使用基于深度学习的计算机视觉模型来支持塑造城市环境的决策。这样的模型预测人们如何从例如它的安全或美丽。但是，深度学习模型的黑盒本质阻碍了城市规划师，以了解哪些景观对象有助于特别高质量或低质量的城市空间感知。这项研究调查了如何使用计算机视觉模型来提取有关人们对城市空间的看法的相关政策信息。为此，我们训练了两个广泛使用的计算机视觉架构。卷积神经网络和变压器，并应用Gradcam（一种众所周知的可解释的AI技术），以突出图像区域对模型的预测很重要。使用这些GradCAM可视化，我们手动注释与模型的感知预测相关的对象。结果，我们能够发现以前研究中用于注释的当前对象检测模型中未表示的新对象。此外，我们的方法论结果表明，变压器架构更适合与GARGCAM技术结合使用。代码可在GitHub上找到。